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Abstract 

A  new  model  of  distributed,  collaborative  information 
evolution  is  emerging.  A?  exemplified  in  Wikipedia,  online 
collaborative  information  repositories  are  being  generated, 
updated,  and  maintained  by  a  large  and  diverse  commu¬ 
nity  of  users.  Issues  concerning  trust  arise  when  content  is 
generated  and  updated  by  diverse  populations.  Since  these 
information  repositories  are  constantly  under  revision,  trust 
determination  is  not  simply  a  static  process.  In  this  paper, 
we  explore  ways  of  utilizing  the  revision  history  of  an  ar¬ 
ticle  to  assess  the  trustworthiness  of  the  article.  We  then 
present  an  experiment  where  we  used  this  revision  history- 
based  trust  model  to  assess  the  trustworthiness  of  a  chain 
of  successive  versions  of  articles  in  Wikipedia  and  evalu¬ 
ated  the  assessments  produced  by  the  model. 

Keywords:  Mining  Revision  History,  Trust  Computa¬ 
tion,  Collaborative  Information  Systems,  Wikipedia 

1  Introduction 

If  users  are  going  to  rely  on  information  they  receive 
from  a  third  party,  they  need  to  have  reasons  to  believe 
that  the  information  is  trustworthy.  In  collaborative  infor¬ 
mation  repositories  such  as  Wikipedia  1 ,  even  if  a  docu¬ 
ment  was  considered  trustworthy  in  the  past,  it  may  not 
still  be  trustworthy  if  it  has  been  changed.  Since  these  in¬ 
formation  repositories  are  constantly  under  revision,  trust 
determination  is  not  simply  a  static  process.  Fortunately, 
collaborative  information  repositories  often  maintain  com¬ 
plete  revision  histories  (change  logs)  of  all  documents.  In 
the  work  reported  in  this  paper,  we  have  explored  the  hy¬ 
pothesis  that  revision  information  can  be  used  to  compute  a 

1  www.wikipedia.com 


measure  of  trustworthiness  of  revised  documents.  Based  on 
that  hypothesis,  we  developed  a  revision  history-based  trust 
model  for  computing  and  tracking  the  trustworthiness  of  the 
documents  in  collaborative  information  repositories.  We 
represent  our  trust  model  in  a  dynamic  Bayesian  network 
(DBN)  because  a  DBN  is  a  powerful  framework  for  model¬ 
ing  processes  that  evolve  dynamically  over  time,  which  in 
our  case,  are  ever  changing  articles. 

Wikipedia  is  a  free  web-based  encyclopedia  and  is  an  in¬ 
teresting  example  of  collaborative  information  repositories 
where  many  people  work  in  a  distributed  manner  to  create 
and  maintain  a  repository  of  shared  content.  In  this  paper, 
we  ground  our  work  in  Wikipedia  because  the  trustworthi¬ 
ness  of  the  articles  in  Wikipedia  is  an  important  and  practi¬ 
cal  issue  as  Wikipedia  continues  to  grow  in  popularity  and 
use.  Additionally,  Wikipedia  provides  rich  and  accessible 
revision  information  to  evaluate  our  approach.  Wikipedia 
users  made  approximately  41  million  revisions,  an  average 
of  12  versions  per  article,  from  July  2002  to  January  2006. 

The  rest  of  our  paper  is  structured  as  follows.  We  intro¬ 
duce  the  concept  and  intuitions  of  revision  trust  in  section 
2.  We  develop  a  dynamic  Bayesian  network  for  our  revi¬ 
sion  history-based  trust  model  in  section  3.  In  section  4,  we 
describe  experimental  results  used  to  evaluate  the  method 
and  discuss  their  implications.  We  discuss  related  work  in 
section  5  and  conclude  our  paper  with  a  discussion  of  future 
work  in  section  6. 

2  Trust  Issues  in  Wikipedia 

Wikipedia  is  a  popular  online  encyclopedia  which  is  col- 
laboratively  written  and  maintained  by  volunteers  world 
wide.  As  of  January  2006,  it  has  more  than  3.3  mil¬ 
lion  articles  in  200  languages  (970,000  articles  in  Eng- 
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lish)  2.  Wikipedia  is  now  among  the  top  30  most  visited 
websites  according  to  the  web  traffic  statistics  provided  by 
Alexa.com. 

As  Wikipedia  continues  to  expand  in  content  and  use, 
issues  concerning  the  trustworthiness  of  information  grow. 
While  recent  studies  (e.g.  [1])  show  that  the  science  articles 
in  Wikipedia  are  generally  trustworthy,  there  have  been  sev¬ 
eral  discoveries  of  inaccurate  or  biased  articles.  Although 
many  approaches  have  been  tried  to  address  the  trust  issue 
in  Wikipedia,  destructive  user  actions  cannot  be  completely 
prevented  due  to  Wikipedia’s  open  editing  policies  that  al¬ 
low  anyone  to  freely  create  and  edit  articles. 

While  manual  monitoring  on  Wikipedia  has  worked 
fairly  well,  we  consider  it  critical  to  build  computational 
trust  models  for  Wikipedia  and  leverage  automated  trust 
management  to  complement  manual  monitoring.  It  is 
preferable  but  manually  infeasible  to  annotate  trustworthi¬ 
ness  at  the  article  level,  because  Wikipedia  as  a  whole  can¬ 
not  be  completely  trusted.  In  addition,  even  if  an  article 
was  considered  tmstworthy  in  the  past,  it  may  not  still  be 
trustworthy  if  it  has  been  changed.  Automated  trust  models 
are  needed  to  monitor  changes  in  trustworthiness  of  articles 
caused  by  revisions. 

2.1  Revision  history-based  trust  model 

This  paper  investigates  automated  trust  computation  us¬ 
ing  the  revision  history  of  an  article.  Throughout  this  paper, 
a  revision  on  an  article  refers  to  the  action  of  editing  the  ar¬ 
ticle  by  an  author.  When  an  article  is  revised,  a  new  version 
of  the  article  is  created  to  archive  the  revised  content.  Thus, 
the  ith  version  of  an  article  is  the  article  after  i  revisions3.  A 
revision  history  of  an  article  is  a  sequence  of  its  versions  or¬ 
dered  by  their  creation  time.  Wikipedia  archives  complete 
revision  history  of  its  articles;  for  example.  Figure  1  shows 
the  first  four  versions  of  the  article  U.S.  National  Forest. 

.  (cur)  (last)  o  08:59,  15  March  2002  Rmhermen 
.  (cur)  (last)  o  20:01 , 4  March  2002  Vicki  Rosenzwe 
environmentalists  (brief  and,  I  hope,  NPOV)) 

.  (cur)  (last)  o  07:58,  4  March  2002  209.255.81 .242 

.  (cur)  (last)  o  20:13,  3  March  2002  24.237.144.197 

Compare  selected  versions 

(Latest  |  Earliest)  View  (previous  50)  (next  50)  (20  |  50  | 

Figure  1.  A  snapshot  of  the  revision  history 
of  the  article  U.S.  National  Forest  in  Wikipedia 


2en.  wikipedia.org/wiki/Wikipedia 

’We  define  the  original  article  to  be  the  0th  version. 


A  revision  history-based  trust  model,  or  simply  revision 
trust  model,  is  built  on  our  hypothesis  that  revision  informa¬ 
tion  can  be  used  to  compute  a  measure  of  trustworthiness: 
trustworthiness  of  the  revised  version  depends  on  the  trust¬ 
worthiness  of  the  previous  version,  the  author  of  the  last 
revision,  and  the  amount  of  text  involved  in  the  last  revi¬ 
sion. 

A  revision  trust  model  may  help  address  a  list  of  inter¬ 
esting  trust  problems  including  the  following: 

•  Article  trust:  tmstworthiness  of  a  version  of  an  article; 

•  Fragment  trust:  tmstworthiness  of  a  fragment  in  a  ver¬ 
sion  of  an  article  (an  article  may  consist  of  fragments 
contributed  by  different  authors); 

•  Author  trust:  trustworthiness  of  an  author;  in  particu¬ 
lar,  author  trust  for  specific  domains.  For  example,  it  is 
possible  to  derive  an  author’s  trustworthiness  with  re¬ 
spect  to  the  domain  of  “Wine”  with  over  500  revisions 
of  the  Wikipedia  article  Wine  or  the  trustworthiness 
of  an  author  with  respect  to  the  domain  of  “Brandy” 
with  150  revisions  of  the  Wikipedia  article  Brandy. 
Such  rich  revision  information  may  provide  enough 
data  for  algorithms  to  infer  domain-specific  trust  to  a 
finer  granularity  than  with  other  approaches. 

As  a  first  step  towards  a  comprehensive  revision  tmst 
model,  we  focus  on  article  trust  in  this  paper,  though  other 
trust  issues,  i.e.  fragment  trust  and  author  trust,  are  also 
under  our  investigation. 

3  A  Revision  Trust  Model  for  article  trust 

We  developed  a  revision  trust  model  to  compute  article 
trust  and  represented  the  model  using  a  dynamic  Bayesian 
network  (DBN). 

There  have  been  extensive  studies  of  the  approaches  to 
quantifying  tmstworthiness  (of  entities).  For  example,  Gol- 
beck  and  Hendler  [2]  defined  a  binary  scale  for  trust  values 
(trusted  or  not  trusted).  In  this  paper,  trust  values  are  de¬ 
fined  over  a  continuous  range  [0, 1],  as  has  been  done  in 
multiple  previous  approaches  (e.g.,  [5]  and  [4]).  We  use  0 
to  represent  complete  untrustworthiness  and  1  to  represent 
complete  trustworthiness.  The  trust  value  of  an  article  is  in¬ 
terpreted  as  the  percentage  of  the  content  in  the  article  that 
is  trustworthy;  for  example,  a  tmst  value  of  0.6  means  60% 
content  of  the  article  is  trustworthy.  Similarly,  an  author 
with  a  trust  value  0.6  means  60%  of  the  content  he  or  she 
writes  is  trustworthy. 

This  trust  representation  might  be  too  simple  to  fully 
capture  the  complexity  of  trustworthiness  in  Wikipedia. 
Nevertheless,  it  has  low  computational  overhead  and  is  in¬ 
tuitive  to  end  users  of  Wikipedia  (when  the  results  are  pre¬ 
sented  visually  to  them).  Our  revision  trust  model  may 


be  extended  to  work  with  other,  potentially  more  complex, 
trust  models. 

3.1  Notation 

We  view  a  revision  as  a  collection  of  deletions,  each  of 
which  removes  some  content  from  an  article,  and  insertions, 
each  of  which  adds  some  content  to  an  article4.  Given  an 
article  with  n  revisions  (thus  with  n  +  1  versions),  we  use 
the  following  notation  throughout  this  paper: 

Vi  refers  to  the  ith  version  of  an  article  which  resulted 
from  a  revision  applied  to  the  previous  version  Vi- 1.  Vq  is 
the  original  article  and  Vn  is  the  final  version.  ty  is  the  trust 
value  of  V. . 

Ai  refers  to  the  author  who  revised  V,  -  \ .  A0  is  the  cre¬ 
ator  of  the  original  article,  t At  is  the  trust  value  of  A,. 

Ii  and  I),  refer  to  the  inserted  content  and  deleted  con¬ 
tent  from  and  to  Vt  by  .4,4  -1  respectively.  If  a;  is  a  piece  of 
text,  then  \x\  denotes  the  size  (i.e.  number  of  words)  of  x. 
It  is  easy  to  see  that  0  <  |/,;|  <  |I^+1|  and  0  <  | Di\  <  \Vi\. 

3.2  A  Dynamic  Bayesian  Network  Trust  Model 

We  use  Dynamic  Bayesian  networks  (DBN)  to  model  the 
evolution  of  article  trust  over  revisions.  The  DBN  is  de¬ 
fined  by  a  pair  ( Bs ,  B0)  where  Bs  is  the  graph  structure  of 
the  network  and  Ba  is  the  set  of  the  network’s  conditional 
density  distributions. 

3.2.1  Graph  structure  Bs 

Figure  2  shows  a  segment  of  Bs  from  version  V)  to  Vi+\. 
The  segment  is  repeated  n  times  in  Bs  (i  from  0  to  n  —  1) 
to  model  a  sequence  of  n  +  1  versions  of  an  article. 


Figure  2.  DBN  segment  from  Vt  to  Vi+i. 

The  state  of  the  DBN  Xi  at  the  ith  revision  is  represented 
as  a  quad  (tyi ,  ,ii,di ),  where  ii  is  the  amount  of  the  in¬ 
sertion,  and  di  is  the  amount  of  the  deletion,  ty.  and  ty,  are 

4In  this  paper,  we  view  an  update  as  equivalent  to  a  deletion  followed 
by  an  insertion,  ignoring  their  subtle  differences. 


continuous  variables  over  [0, 1],  while  it  and  di  are  contin¬ 
uous  variables  over  [0, 00].  Our  DBN  satisfies  the  Markov 
property:  /(Xi+1|Xi,Xi_1,...1X0)  =  /(X,+1|Xi). 

Additionally,  the  solid  arrows  in  Figure  2  encode  the  de¬ 
pendency  relationships  in  the  DBN  segment.  The  trustwor¬ 
thiness  of  an  article  version  is  dependent  on  the  trustwor¬ 
thiness  of  the  previous  version  and  the  author  of  the  last 
revision,  the  amount  of  the  insertion,  and  the  amount  of  the 
deletion. 

There  may  be  dependencies  between  i,  and  ty,  and 
tAi+l,  and  between  di  and  ty.  and  ,  as  indicated  by 
the  dotted  arrows  in  Figure  2.  For  example,  a  trustworthy 
author  is  likely  to  make  a  large  amount  of  changes  to  a  very 
untrustworthy  article.  This  paper  assumes  independencies 
between  ii,  di  and  tyi,  tAi+1 ■  Our  assumption  might  be  an 
issue  to  debate;  nevertheless,  ii  and  di  are  also  dependent 
on  other  important  factors,  such  as  the  interest  an  author 
has  in  an  article.  Since  these  factors  are  not  modeled  in  the 
DBN,  deciding  the  conditional  density  functions  between 
ii,  di  and  fy;,  tAi+1  can  be  difficult  and  less  useful.  Al¬ 
though  we  are  still  investigating  approaches  to  remove  the 
independence  assumption,  good  results  have  been  observed 
in  our  experiments  even  with  this  assumption. 

We  seek  to  determine  the  posterior  density  distribu¬ 
tion  of  f(tyi+1).  Given  the  dependencies  in  the  graph 
structure  Bs,  Ba  is  fully  characterized  by  f{ty0\tA0)  and 
f  1  \i~Vi  >  ^.4,41 5  ti,  di) . 

3.2.2  f{tVo\tAo) 

The  trustworthiness  of  the  original  article  is  only  dependent 
on  its  author  in  our  model.  If  we  assume  the  dependency 
between  ty0  and  t,40  is  deterministic,  the  conditional  prob¬ 
ability  is:  P(tVo  =  a0\tAo  =  a0)  =  1. 

That  is,  if  an  author  is  0.8  trustworthy,  then  an  article 
written  by  that  author  is  0.8  trustworthy  with  probability 
1.  Nevertheless,  in  reality,  that  article  is  very  unlikely  to 
be  precisely  0.8  trustworthy.  For  example,  the  trustworthi¬ 
ness  of  the  article  mentioned  above  can  be  in  a  range  of 
0.75  —  0.85  with  high  probability.  The  uncertainty  in  the 
trustworthiness  is  caused  by  uncertain  factors  such  as  the 
context  where  an  author  writes  an  article. 

A  common  approach  is  to  assume  normality  and  model 
conditional  density  functions  with  Gaussian  distributions. 
In  this  paper,  we  chose  beta  distributions  because  trust  vari¬ 
ables  are  defined  in  the  range  [0, 1]  where  beta  distributions 
are  also  defined.  Nevertheless,  we  do  not  view  revisions 
as  independent  events,  even  though  beta  distribution  is  nor¬ 
mally  associated  with  binomial  process. 

3.2.3  Beta  distributions 

The  beta  distributions  are  a  family  of  distributions  with  two 
parameters  a  and  (3. 


beta(p\a,P)  =  ^^p)Pa  \l-p)0  1,  where  B(x,  y) 

is  the  beta  function:  B(x,  y)  =  fg  tx~1(  1  — 

beta(p\a,  (3)  can  take  on  different  shapes  depending  on 
the  values  of  a  and  f3,  as  depicted  in  Figure  3.  When 
a,  (3  >  1,  the  curve  is  a  desired  unimodal  for  modeling  the 
uncertainty  in  the  trust  distribution.  We  chose  a,  /3  >  10 
in  this  work  as  we  make  a  simplifying  assumption  that  the 
variance  a  of  the  distribution  should  be  neither  too  small 
nor  too  large  when  the  mean  //  is  close  to  0.5.  On  the  other 
hand,  if  //  is  close  to  0  or  1,  a  does  not  make  much  differ¬ 
ence  in  the  choices  of  a  and  f3  because  a  is  bounded  by  a 
very  small  value  /z(l  —  //.).  Given  this  constraint  and  the  fact 
that  the  mean  of  beta(p\a,  (3)  is  p  =  ^g,  we  have 


a  =  10,{3  =  ^;or 

10m  Ft  _ 


=  =  lo¬ 


ci) 


Probability  Density  Function  of  Beta  Distribution 


Figure  3.  Shapes  of  beta  distributions  with  dif¬ 
ferent  a  and  (3.  The  mean  of  beta{p\a,  (3)  is  y  = 
and  the  variance  is  a 


afj 


a+0 


(a+/3)2((a+/3+l)' 


Given  Equation  1,  we  only  need  to  compute  im+i  to  fully 
determine  beta(p\ai+i,  A+i). 

In  our  approach,  the  trustworthy  portion  of  Fi+1  is  the 
trustworthy  portion  of  Vt  plus  the  trustworthy  insertion  of 
Ai+ 1  minus  the  trustworthy  portion  that  Ai+i  incorrectly 
removed  from  V, . 

We  know  that  t\Vi  \  of  V.  is  trustworthy  and  (1  —  t)\Vi\  is 
untrustworthy.  Additionally,  since  ta  =  ai+i.  we  expect 
ai+i\Ii\  of  Ii  to  be  trustworthy  and  (1  —  aj+i)|/j|  to  be 
untrustworthy. 

The  consequences  of  the  deletions  are  complicated.  In 
this  paper,  we  consider  the  following  four  scenarios  about 
deletions5 : 

•  (1  -  ai+i)|A|  <  t\Vi\  and  ai+1|A|  <  (1  -  t)\Vi\. 
Ai+ 1  expects  to  (incorrectly)  remove  (1  —  aj+ 1  )|A| 
of  the  trustworthy  portion  of  Vi.  In  this  case,  since  this 
amount  removed  is  no  more  than  the  amount  of  the  ex¬ 
isting  trustworthy  portion  in  Vi  (i.e.  f|Vi|),  Ai+\  actu¬ 
ally  removes  (1  —  aj+i)|  A|  of  the  trustworthy  portion 
of  Vp, 


•  (1  -  ai+i)|A|  >  t\Vi\  and  ai+i|A|  <  (1  -  t)\Vi\. 

Ai+ 1  (incorrectly)  removes  all  the  trustworthy  portion 
of  Vp, 


•  (1  -  ai+i)|A|  <  t\Vi\  and  ai+i|A|  >  (1  -  t)W\. 
Ai+i  (correctly)  removes  all  the  untrustworthy  portion 
of  V.  (i.e.  (1  —  t)|Vi|);  thus,  Ai+i  has  to  remove  an 
additional  A|  —  (1  —  t)|Vj|  amount  of  the  trust¬ 
worthy  portion  of  V,.  The  total  amount  of  the  trust¬ 
worthy  portion  that  Ai+i  removes  from  Vi  is  thereby 

(1  -  ai+1)|A|  +  (ai+tlAI  -  (1  -  t)\V\). 


•  (1  -  af+i)|A|  >  t\Vi\  and  ai+i|A|  >  (1  -  i)|Vj|. 

This  case  is  impossible  because  A  I  <  N- 


The  distribution  of  ty 0  given  t.A„  is  therefore 

f{ty o \tA0  =  «o)  =  beta{p\a0,  (30)  (2) 

We  choose  yo  =  ao ,  then  ao  and  (3$  can  be  determined 
by  Equation  1 . 

3.2.4  f  (tVi+i  \tVi  }  i  di) 

We  now  consider  the  conditional  density  distribution 
of  fy  in  the  DBN,  for  0  <  i  <  n  —  1;  similarly  to 
f(ty0 1 f  a0),  we  assume  that  these  distributions  are  also  beta 
distributions: 

f  (tVi+x  I ^Vi  —  t0Ai+i  —  |  >  di  —  |  Di  | ) 

=  beta(p\ai+i,  pi+i) 

(3) 


In  summary,  the  mean  of  beta(p\ai+i,  /3i+1),  is 

the  size  of  the  trustworthy  portion  in  Vj+i  divided  by  the 
total  size  of  Vj+i,  where  |Vi+i|  =  \Vi\  +  \Ii\  —  \Di\.  We 
combine  the  above  four  cases  into  Equation  4. 

Hi+ 1  =  {t\V\  +ai+i\Ii\  -  min((l  -  ai+i)|A|,i|Vi]) 

-  max(ai+i|A|  -  (1  -  t)\Vi\,0)}/\Vi+i\ 

(4) 

Given  in  Equation  4,  oti+i  and  (3i+\  are  determined 
by  Equation  1,  and  thus  the  conditional  distribution  in  Equa¬ 
tion  3  is  determined.  Our  DBN  is  now  complete  with  both 
Bs  and  B0  fully  defined. 


5  Our  method  is  not  the  only  approach  to  determine  the  trustworthy  por¬ 
tion  that  Ai+i  incorrectly  removed  from  Vi. 


4  Experiments 

We  now  describe  the  experiments  and  their  results.  The 
inference  computations  in  our  DBN  were  carried  out  us¬ 
ing  the  freely  available  software  BUGS  (Bayesian  inference 
Using  Gibbs  Sampling  [11]).  BUGS  is  a  software  pack¬ 
age  for  BN  inference  using  the  Markov  Chain  Monte  Carlo 
(MCMC)  method. 

4.1  Experiment  Settings 

In  order  to  evaluate  our  trust  model  for  computing  article 
trust,  we  collected  a  set  of  English  articles  from  the  Geog¬ 
raphy  category  in  Wikipedia  in  January  2006.  We  chose 
articles  from  the  same  category  so  that  their  trustworthiness 
values  could  be  viewed  as  comparable. 

Evaluation  of  trust  algorithms  in  Wikipedia  is  difficult 
because  the  trustworthiness  of  Wikipedia  articles  is  not  ex¬ 
plicitly  asserted  and  is  subject  to  personal  opinions.  To  ob¬ 
tain  the  reference  trust  value  for  each  article,  we  consider 
three  manually  classified  groups  of  articles  in  Wikipedia: 

•  featured  articles6 7  which  are  considered  highly  trust¬ 
worthy  in  the  Wikipedia  community  because  they  have 
been  thoroughly  reviewed  for  style,  prose,  complete¬ 
ness,  accuracy  and  neutrality; 

•  clean-up  articles1  which  are  considered  untrustworthy 
because  they  have  been  marked  for  major  revision  by 
Wikipedia  authors; 

•  normal  articles  which  are  the  remaining  articles. 

Our  data  set  consisted  of  50  featured  articles,  50  clean-up 
articles,  and  768  normal  articles  (a  total  of  40450  revisions). 
The  number  of  featured  articles  and  clean-up  articles  we 
considered  in  the  experiments  is  relative  small  because  only 
0.1%  of  Wikipedia  articles  are  featured  articles  and  1.3% 
are  clean-up  articles.  In  particular,  there  are  less  than  80 
featured  articles  in  the  Geography  category. 

Author  trust  is  synthesized  based  on  the  background 
knowledge  from  Wikipedia.  Although  there  are  no  ex¬ 
plicit  trust  values  associated  with  the  authors  in  Wikipedia, 
we  can  provide  coarse  approximations  based  on  the  edit¬ 
ing  privileges  of  the  authors.  Currently,  Wikipedia  sup¬ 
ports  four  levels  of  authorship:  administrators  (including 
stewards  and  developers),  registered  authors,  anonymous 
authors,  and  blocked  authors,  with  each  level  having  de¬ 
creasing  editing  powers  and  trustworthiness.  We  thereby 
use  the  following  beta  distributions  to  approximate  the  trust¬ 
worthiness  of  the  authors  :  6eto(p|190, 10)  for  administra¬ 
tors,  beta(p 1 23, 10)  for  registered  authors,  beta(p 1 15, 10) 

6en.wikipedia.org/wiki/Wikipedia:FeaturecLarticles 

7en.  wikipedia.org/wiki/Clean-up 


for  anonymous  authors,  and  beta{p\  10, 190)  for  blocked  au¬ 
thors.  The  means  of  the  distributions  are  0.95,  0.7,  0.6,  and 
0.05  respectively.  The  exact  values  are  chosen  rather  arbi¬ 
trarily;  however,  the  relative  differentials  between  the  trust 
values  reflect  our  assessment  of  the  trustworthiness  of  the 
authors  with  different  editing  privileges. 

The  amount  of  insertion  and  deletion  of  each  revision 
can  be  calculated  by  comparing  consecutive  article  ver¬ 
sions  using  a  dijf  algorithm.  Our  implementation  of  diff 
was  based  on  the  well  known  longest  common  subsequence 
algorithm  [9];  and  the  differences  between  versions  were 
counted  at  word  level. 


Table  1 .  The  statistics  of  data  set 


featured 

articles 

clean-up 

articles 

normal 

articles 

Average  number 
of  revisions 

726 

56 

27 

Average  percentage 
of  administrators 

29.4% 

12.5% 

28.4% 

Average  percentage 
of  registered  authors 

45.2% 

56.6% 

52.1% 

Average  percentage 
of  anonymous  authors 

24.1% 

29.6% 

18.7% 

Average  percentage 
of  blocked  authors 

1.3% 

1.3% 

0.8% 

Average  size  of 
the  final  version 

3385 

524 

274 

Average  percentage  of 
insertion  per  revision 

2.5% 

6.3% 

6.2% 

Average  percentage  of 
deletion  per  revision 

2.1% 

4.7% 

2.4% 

Table  1  shows  the  statistics  of  the  data  set.  Featured 
articles  have  far  more  revisions  than  clean-up  articles  and 
normal  articles,  while  clean-up  articles  have  the  lowest  per¬ 
centage  of  administrator  authors.  Though  the  number  of  re¬ 
visions  and  the  trustworthiness  of  authors  are  the  two  most 
important  factors  in  determining  the  trustworthiness  of  an 
article,  other  factors,  such  as  the  amount  of  the  changes  and 
the  order  of  the  revisions  are  also  important.  It  is  interesting 
to  note  that  normal  articles  have  the  lowest  number  of  revi¬ 
sions,  which  shows  that  both  featured  articles  and  clean-up 
articles  receive  more  attentions  from  Wikipedia  authors. 

4.2  Evaluation  and  Discussion 

We  defined  and  ran  our  trust  model  in  BUGS  and  ob¬ 
tained  posterior  density  distributions  /(ty. )  for  article  ver¬ 
sions  in  our  data  set.  For  example.  Figure  4  shows  the 
trust  density  distributions  of  Vq  and  T43  of  the  article  U.S. 


National  Forest  8.  U.S.  National  Forest  was  created  by 
an  anonymous  author,  thus  the  distribution  of  ty0  is  simi¬ 
lar  to  the  trust  distribution  of  anonymous  authors.  The  fi¬ 
nal  version  was  highly  trustworthy  after  43  revisions.  In 
most  cases,  the  trustworthiness  measure  of  article  versions 
is  increasingly  precise  with  more  revisions;  for  example,  as 
shown  in  Figure  4,  the  variance  of  f(tv43)  is  much  smaller 
than  that  of  f(tv0)', 


(a)  Distribution  of  t,  (b)  Distribution  of  t,. 

*  0  *43 


Figure  4.  Trust  density  distributions  of  the 
first  version  and  the  final  version  of  the  ar¬ 
ticle  U.S.  National  Forest. 


We  are  most  interested  in  the  mean  of  f(tv„ ).  which  we 
refer  as  JIJJ  9.  JIJJ  indicates  the  trustworthiness  of  the  lat¬ 
est  version  of  an  article.  The  results  are  summarized  in 
Table  2.  Featured  articles  have  the  highest  average  of  /ir, 
and  the  lowest  variance,  while  clean-up  articles  are  the  op¬ 
posite.  Note  that  even  though  several  simplifying  assump¬ 
tions  were  made  in  this  work,  our  model  showed  significant 
differences  of  trustworthiness  between  featured  articles  and 
clean-up  articles. 


Table  2.  The  average  and  variance  of  jrj  of  50 
featured  articles,  50  clean-up  articles  and  768 
normal  articles. 


featured 

articles 

clean-up 

articles 

normal 

articles 

Average  of  pn 

0.885 

0.768 

0.808 

Variance  of  JlJJ 

0.011 

0.019 

0.016 

The  values  in  Table  2  are  the  trust  values  of  the  final 
versions  (i.e.  the  most  recent  versions)  of  featured  articles 
and  clean-up  articles.  The  average  Jl  when  featured  articles 
were  initially  approved  is  0.903,  just  slightly  higher  than  the 
current  average  0.885.  On  the  contrary,  the  average  Jl  when 
clean-up  articles  were  marked  is  0.739,  lower  than  the  cur¬ 
rent  value  0.768.  We  found  that  six  clean-up  articles  (out  of 
50)  had  been  dramatically  improved  since  they  were  marked 
as  clean-up  articles.  For  example,  Jl  of  article  Victoria  Park 

8en.wikipedia.org/w/index.php?titfe=U.S. -National  .Forest 
is  the  mean  of  the  posterior  density  distribution 
while  in  Equation  4,  //,,,  is  the  mean  of  the  conditional  distribution 
f{Wn  |*V„_i  i  ,  in-i,  dn-i). 


(Hartlepool)  jumped  from  0.705  to  0.884  within  a  month.  It 
seems  that  typically  there  are  small  changes  of  trustworthi¬ 
ness  in  featured  articles  after  they  became  featured  articles, 
while  clean-up  articles  are  being  continuously  improved. 

We  developed  a  classifier  based  on  the  aforementioned 
50  featured  articles  and  50  clean-up  articles  in  Table  2  to 
separate  featured  articles  and  clean-up  articles.  The  train¬ 
ing  set  contains  100  pairs  (x,  y),  where  x  is  the  trust  value 
of  an  article  and  y  is  its  class.  In  this  data  set,  we  only  con¬ 
sider  the  revision  history  of  a  featured  article  (or  a  clean-up 
article)  up  to  the  point  where  it  was  initially  approved  by 
Wikipedia  authors.  We  do  not  use  the  complete  revision  his¬ 
tory  which  includes  the  most  recent  article  versions  whose 
trustworthiness  may  be  unknown.  Since  we  use  the  trust 
value  to  predict  a  class  that  an  article  belongs  to,  the  learned 
rule  for  featured  article  is:  x  >  0.842.  Thus  in  practice  we 
consider  an  article  trustworthy  if  its  trust  value  is  higher 
than  0.842.  A  test  size  of  200  new  articles  (48805  article 
revisions)  was  evaluated.  The  percentage  for  correctly  pre¬ 
dicting  featured  articles  is  82%  and  that  for  clean-up  articles 
is  84%. 

We  studied  citation-based  trust  in  [8],  Citation-based 
trust  algorithms  are  a  family  of  algorithms  that  derive  trust 
based  on  the  citation  relationships  among  articles.  For  ex¬ 
ample,  a  well-cited  article  may  be  more  trustworthy  than  an 
article  that  has  no  citation.  We  showed  that  citation-based 
trust  algorithms  may  not  be  very  effective  for  computing 
trustworthiness  of  assertions  in  an  aggregated  knowledge 
repository  such  as  Wikipedia.  In  particular,  the  percent¬ 
age  for  correctly  predicting  featured  articles  is  46%  and 
that  for  clean-up  articles  is  54%  based  on  a  classifier  de¬ 
veloped  from  a  citation-based  trust  algorithm.  Clearly,  re¬ 
vision  history-based  trust  algorithm  significantly  improves 
the  accuracy  of  trust  computation. 

Based  on  manual  inspection,  our  model  appears  correct 
in  estimating  the  changes  of  trustworthiness  caused  by  ma¬ 
jor  revision  events  (e.g.  a  revision  with  more  than  10%  in¬ 
sertion  and/or  deletion).  For  example,  in  Figure  5,  we  ex¬ 
amine  every  JTJ  from  the  first  version  to  the  final  version  of 
U.S.  National  Forest.  We  manually  identified  six  major  re¬ 
visions  from  the  revision  history  of  the  article  and  depicted 
these  events  in  Figure  5.  Our  trust  model  successfully  esti¬ 
mated  the  consequences  of  five  events  (which  are  depicted 
in  white  boxes).  The  only  exception  is  event  four,  where 
our  model  predicted  a  decrease  of  trustworthiness  due  to  a 
15%  insertion  made  by  a  blocked  author.  Nevertheless,  our 
manual  analysis  showed  that  this  author’s  insertion  here  was 
genuine.  Our  results  may  be  improved  by  developing  author 
trust  models  for  modeling  complicated  author  trust  behav¬ 
iors.  For  example,  lack  of  expertise  (blocked  status)  in  one 
area  does  not  mean  a  lack  of  expertise  (blocked  status)  in 
another. 

Additionally,  we  have  the  following  observations:  (1) 
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revision  history. 

Other  related  work  includes  mining  web  logs  (e.g.,  [12]) 
and  mining  software  revision  history  (e.g.,  [7]).  Most  such 
applications  were  built  on  mining  association  rules.  How¬ 
ever,  association  rules  or  simple  revision  parameters  (such 
the  number  of  revisions)  are  not  very  useful  in  computing 
and  tracking  trustworthiness  of  articles  that  are  under  con¬ 
stant  changes.  For  example,  a  featured  article  could  become 
untrustworthy  if  it  has  been  changed  despite  the  fact  that  the 
number  of  revisions  is  monotonically  increasing. 

6  Conclusions 


Article  Version  V. 

i 

Figure  5.  Changes  in  trustworthiness  of  all 
versions  of  U.S.  National  Forest.  These  events 
except  event  four  are  consistent  with  the  re¬ 
sults  of  our  model. 


The  changes  of  trustworthiness  are  rather  smooth  unless  a 
major  revision  occurs;  (2)  In  event  5,  a  large  drop  in  trust¬ 
worthiness  was  caused  by  a  mass  deletion,  which  was  sub¬ 
sequently  reverted  by  an  administer  in  event  6.  The  trust¬ 
worthiness  of  the  reverted  version  might  be  considered  to 
be  the  same  as  the  version  before  the  vandalism  occurred, 
but  since  the  revision  was  performed  by  an  administrator, 
the  trustworthiness  was  higher  in  our  model. 

5  Related  Work 

Lih  [6]  studied  the  correlation  between  the  numbers  of 
revisions  and  unique  authors  to  Wikipedia  articles  and  the 
quality  of  these  articles.  Voss  [14]  conducted  a  compre¬ 
hensive  analysis  on  various  aspects  of  Wikipedia  articles. 
Viegas  et  al.  [13]  presented  a  tool  that  visualized  revi¬ 
sion  history  flow  and  through  which  they  revealed  some  in¬ 
teresting  revision  patterns  in  Wikipedia.  While  qualitative 
studies  are  important,  we  believe  our  approach  is  the  first 
computational  trust  model  utilizing  revision  history  data  in 
Wikipedia. 

Trust  in  P2P  and  social  networks  has  been  extensively 
studied  in  recent  years  [5],  [4]  and  [3],  Unlike  the  typi¬ 
cal  social  networks,  the  trustworthiness  of  authors  and  arti¬ 
cles  are  not  explicitly  asserted  in  Wikipedia.  Revision  trust 
is  substantially  different  from  those  approaches  which  are 
based  on  the  transitivity  property  of  trust. 

While  our  current  use  of  Bayesian  networks  is  mod¬ 
est,  the  reader  can  refer  to  [10]  for  more  background  on 
Bayesian  networks  and  DBNs  and  envision  future  exten¬ 
sions  to  our  work,  e.g.,  learning  a,  and  Pt  parameters  from 


Trust  in  collaborative  information  repositories  is  becom¬ 
ing  increasingly  important  as  people  are  relying  more  on 
online  information  and  are  more  actively  participating  in 
online  collaborations.  In  this  paper,  we  made  the  following 
contributions  towards  the  understanding  and  computing  of 
trust  in  collaborative  information  repositories,  in  particular, 
Wikipedia.  We  introduced  the  concept  of  revision  history- 
based  trust  and  developed  a  dynamic  Bayesian  network  trust 
model  that  utilized  rich  revision  information  in  Wikipedia. 
Our  experiments  showed  promising  results,  even  though  we 
made  several  simplifying  assumptions  in  this  work.  We 
showed  an  evaluation  method  in  Wikipedia  based  on  its  fea¬ 
ture  articles,  clean-up  articles  and  the  levels  of  author  edit¬ 
ing  privileges.  Our  work  provided  a  methodology  for  com¬ 
paring  and  evaluating  future  computational  trust  algorithms. 

Based  on  our  DBN  model,  we  believe  the  reasons 
for  Wikipedia  being  generally  trustworthy  are:  (1)  most 
Wikipedia  authors  seem  to  have  good  intentions  (there  are 
only  1.3%  blocked  authors);  (2)  Wikipedia  administrators 
have  the  responsibility  and  authority  to  settle  disputes,  pre¬ 
vent  vandalism,  and  block  inappropriate  authors.  While 
there  are  a  small  number  of  administrators  (0.09%),  they 
have  made  much  larger  contributions  to  Wikipedia,  for  ex¬ 
ample,  29.4%  revisions  of  featured  articles  were  made  by 
administrators  in  our  experiments,  according  to  Table  1;  (3) 
Wikipedia  maintains  a  complete  revision  history  of  articles 
from  which  a  previous  content  modification  can  be  easily 
reverted. 

The  benefits  of  revision  trust  to  Wikipedia  users  are  sig¬ 
nificant.  Visualization  of  the  computed  trust  values  may 
help  users  to  decide  what  information  they  should  trust. 
Users  may  also  have  the  option  to  view  the  most  trustwor¬ 
thy  version  of  an  article,  in  addition  to  the  most  recent  one. 
Furthermore,  revision  trust  can  improve  Wikipedia’s  qual¬ 
ity  control  process;  for  example,  our  model  provides  an  ap¬ 
pealing  approach  to  monitoring  changes  in  trustworthiness 
and  thereby  providing  timely  notifications  of  vandalism  and 
other  forms  of  malicious  content  modifications. 

Our  work  may  be  extended  in  several  directions:  (1)  The 
article  trust  model  may  be  refined  and  improved;  we  ex- 


pect  better  performance  when  the  simplifying  assumptions 
in  our  model  are  removed;  (2)  Revision  trust  may  help  to 
solve  many  difficult  trust  issues,  e.g.,  fragment  trust  and  au¬ 
thor  trust;  (3)  Though  we  focused  on  computing  trust  in 
Wikipedia,  it  is  interesting  to  extend  revision  trust  to  other 
collaborative  systems  with  rich  revision  information. 
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